cc @tlimoncelli

Also, I have already begun running intergration tests under my own account's GitHub Action here: SukkaW#1

https://github.com/SukkaW/dnscontrol/actions/runs/19867709318/job/56938063153?pr=1

Is it normal to be stuck at this step? Doesn't feel like the test has begun yet, it appears to be stuck at go install (which really should not happen).

@tlimoncelli OK, I figured out what's actually going on. Turns out that even w/o the pager test, the rest of the intergration test still creates more than 100 records and just hits Vercel's 100 creations per hour rate limit.

Thanks for all the research. I really appreciate you taking the time to investigate this.

I agree with your conclusion. This integration test just does too much. There are a couple ways forward: fix it or ignore it.

The "fix it" route would mean detecting the error, pausing, and trying again. Often APIs include headers that tell you how long to wait, which is super. Otherwise, retrying every minute is generally fine. (add -timeout 3h to the test lets it run longer than the default 10 minutes)

The "ignore it" route has a few benefits. One is that ... less work. Of course, the downside is that if something breaks, we might not know about it right away. That's probably ok. You'd be surprised at how rarely that has happened for other providers. Generally companies are good about not changing APIs.

I'm considering adding a feature where tests mark themselves as "long" or "short" and there's a good chance that running only the short tests will be fewer than 100 changes (50 adds, 50 deletes). However that's months away (maybe a year) in the roadmap. My point is that "ignore it" has the benefit that if we wait long enough the problem may go away when the "short" tests feature arrives.

Another route is to ask Vercel for an exception. Explain the situation and ask if they have a solution. Maybe they have a test environment with no limits, or can grant this test account a higher limit if we promise to only use it for these occasional tests. Vendors have raised limits for us in the past.

I'm fine with all these options. Let me know your preference.

The "fix it" route would mean detecting the error, pausing, and trying again. Often APIs include headers that tell you how long to wait, which is super. Otherwise, retrying every minute is generally fine. (add -timeout 3h to the test lets it run longer than the default 10 minutes)

Actually, I have already implemented that, and the provider will actually wait for the rate limit quota resets. It is just that we need to wait for an entire hour before continuing, while the default test timeout is 30 minutes.

The "ignore it" route has a few benefits. One is that ... less work. Of course, the downside is that if something breaks, we might not know about it right away. That's probably ok. You'd be surprised at how rarely that has happened for other providers. Generally companies are good about not changing APIs.

The pager test has already been skipped, but other tests accumulated more than 100 records that resulted in exhausting rate limits.

Another route is to ask Vercel for an exception. Explain the situation and ask if they have a solution. Maybe they have a test environment with no limits, or can grant this test account a higher limit if we promise to only use it for these occasional tests. Vendors have raised limits for us in the past.

Lemme get in contact with Vercel to see if it helps.

@tlimoncelli Just an update. By running intergration tests locally (and luckily spotted a few bugs #3887), I confirm that we can at least finish test group 1 to 50 within the 100 limit. If we skip all IGNORE* series tests for Vercel, we might actually be able to finish tests w/o increasing limits.

@tlimoncelli Just an update. By running intergration tests locally (and luckily spotted a few bugs #3887), I confirm that we can at least finish test group 1 to 50 within the 100 limit. If we skip all IGNORE* series tests for Vercel, we might actually be able to finish tests w/o increasing limits.

Sounds good!

P.S. The IGNORE() tests tests DNSControl more-so than the provider. If they complete successfully once, they'll probably pass forever.

Update: Vercel rejects my request to increase the rate limit to the test account. In fact, no human has ever read my email. It was an automated response with "Hobby Plan does not have any support".